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ABSTRACT 

Demands for online graduate degrees have increased pressure on universities to launch 
web degrees quickly and, at times, without attending to their quality. Scarce research exists 
identifying what evaluation activities are being done by science, technology, engineering, and 
mathematics (STEM) online graduate degree programs that are accustomed to accreditation 
procedures for their undergraduate degrees. This case study investigated three established online, 
graduate degree programs in agriculture and engineering at a prominent, Midwestern university, 
and identified evaluation practices used in these programs. Findings suggested that even though 
they are fully accredited in the undergraduate programs, most of the evaluation of their online 
degree program is informal, with three key themes emerging, including the use of: (a) informal 
feedback from the students and faculty, (b) student satisfaction surveys, (c) student grades and 
performance information. 

There were several issues reported in using these three strategies to evaluate quality, 
including lacking structured collection and reporting mechanisms, differing implementation 
levels in traditional and online courses, varying availability of data and student quality, and 
lacking fidelity of information delivery and access. The findings suggest that the use of these 
informal evaluative processes have implications for evaluating STEM graduate degree programs 
online. 

Keywords: online graduate degrees. Science, Technology, Engineering, and Mathematics 
(STEM), higher education, evaluation, case study 
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INTRODUCTION 

Online learning practices have become ubiquitous in higher education, with 89 percent of 
public institutions using networked computing and communication technologies to serve the 
estimated 3 million U.S. higher education students who have taken at least one online course 
since 2007 (Allen & Seaman, 2012; Borgman et ah, 2008; Parsad & Lewis, 2008; Sprague, 
Maddux, Ferdig, & Albion, 2007; U.S. Department of Education, Office of Innovation and 
Improvement, 2008). Demands to provide online learning have been further augmented by 
economic factors, such as the increase in gas prices and layoffs and the increased interest, 
promotion, and delivery of education on a global scale (Allen & Seaman, 2008; Allen & Seaman, 
2012; Sprague, Maddux, & Ferdig, 2007). While online education has been present in nearly all 
public universities since 2000, the way to rigorously evaluate the question of, “How can a 
teaching/leaming process that deviates so markedly from what has been practiced for hundreds 
of years embody quality education?” (Phipps & Merisotis, 2000, p.7) still remains unanswered, 
particularly for graduate degree programs offered entirely online and from fields in science, 
technology, engineering, and mathematics (STEM) which have a historical preference for 
program quality monitoring and accreditation (ABET, 2009). 

Therefore, this study undertook a multiple case study approach using a holistic approach 
to probe just exactly how three prominent STEM programs with entirely online graduate degree 
programs are evaluating their programs (Yin, 2002). The study centered on the following 
research question: 

How are the Midwestern University online graduate programs from the various STEM 

fields discerning the quality of their programs? How can a deeper analysis of these three 

established online programs inform the evaluation practices of other similar programs? 

REVIEW OF THE LITERATURE 

Online learning is defined as classes with at least 80% of the content delivery done online 
(Allen & Seaman, 2012). Ideally, these online courses are characterized by distributed control, 
with the instructor sharing control with the students utilizing web-based, multiple media to 
facilitate knowledge building and interaction via synchronous and asynchronous communication 
tools(Collison, Tinker, Elbaum, & Haavind, 2000; Dabbagh & Bannan-Ritland, 2005). However, 
defining the quality of these ventures is harder to do. “Quality” is a conflict-ridden term because 
of its amorphous nature. Robert M. Pirsig in his book Zen and the Art of Motorcycle 
Maintenance: An Inquiry into Values describes this struggle: “I think that there is such a thing as 
Quality, but that as soon as you try to define it, something goes haywire.. .But even though 
Quality cannot be defined, you know what Quality is” (Pirsig, 2005). Nevertheless, definitions 
have been attributed to “quality,” and this becomes especially important in evaluation of online 
settings. Scholars on evaluation theory, Robert Stake and Thomas Schwandt, assert that “Quality 
is a broad term that encompasses notions of merit, worth, and significance” (Stake & Schwandt, 
2006). Moreover, establishing common indicators of quality and ways to probe this is a necessity 
because “historically, quality holds the key to academic acceptance” (Larreamendy-Joerns & 
Leinhardt, 2006, p. 579). This acceptance is especially important to online learning environments 
because of the trepidation among some in higher education that online education is jeopardizing 
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the quality of higher education (Burbules & Callister, 2000; Larreamendy-Joerns & Leinhardt, 
2006; Rovai, 2003). 

While higher education quality assurance has received increased scrutiny after reports 
like A Test of Leadership: Charting the Future of U.S. Higher Education from the Commission 
on the Future of Higher Education (U.S. Department of Education, 2006) and the additions to the 
Higher Education Opportunity Act (US Department of Education, 2008), the assessment of 
undergraduate and graduate courses has consisted of results drawn primarily from surveys of 
student satisfaction and a self-assessment of learning via post-participation surveys administered 
immediately following course participation or upon degree completion (Aragon, 2003; Johnson 
& Aragon, 2003). Consequently, monitoring the quality and establishing quality indicators for 
STEM graduate programs falls squarely on the institutions and often the administrators of the 
traditional and online graduate programs. Therefore, it was important investigate the empirical 
reporting of traditional, online, and STEM quality monitoring sources in the literature before 
conducting this study in order to understand the larger context. 

First, in traditional program quality monitoring, extensive literature was reviewed. In 
light of the desire for accountability standards for program quality monitoring, it was particularly 
important for this study to consider what universities use to assess themselves traditionally 
beyond reputational rankings used by popular media like the U.S. News & World Report. In an 
attempt to move away from the problems of the “soft” criterion from measures like these of 
scholarly quality, the National Research Council (NRC) completed their 2009 survey on 
assessing research-doctorate programs (Ostriker, Kuh, & Voytuk, 2003). The study included an 
all-faculty survey from which a random sample was drawn and rated to represent data from their 
particular fields, including STEM departments. The NRC study was particularly relevant to this 
research because the Midwest University participated in the 1995 and the 2009 NRC. While the 
NRC study was aimed at doctorate programs, the quality indicators used to generate data on 
reputation, faculty research, and student experiences are often used in studies of quality in 
graduate education regardless of the level of degree (Brooks, 2005; Perry, 1994).Some literature 
from agriculture and engineering suggests that reputational and quality rankings are more 
dependent on the highest degree offered by the institution (that being the Ph.D. at Midwest 
University) rather than rankings of undergraduate or Master’s degree programs (Perry, 1994; 
Zhang, 2005) 

Next, for online program quality monitoring, the establishment of quality indicators for 
online programs is especially important for legitimacy and moving past the “no significance 
difference” debate (Phipps & Merisotis, 2000; Ruhe & Zumbo, 2008; Seok, 2007). Prior 
literature suggests that norms for online programs must be created before outcomes can be 
asserted, and this is commonly done by using quality indicators or benchmarks from an 
institution’s traditional learning programs or using established quality frameworks for online 
learning environments (Larreamendy-Joerns & Leinhardt, 2006; Rovai, 2003). Currently, there 
are no formal standards or quality indicators in the field of traditional or online learning in 
graduate studies, but a number of quality frameworks commonly referenced in the literature have 
been used to monitor quality of online learning. The most commonly referenced frameworks in 
the literature include: (a) Quality on the Line from Institute for Higher Education Policy (Phipps 
& Merisotis, 2000), (b) Quality Matters from the University of Maryland (MarylandOnline, 
2008; Sener & Shattuck, 2006), and (c) The Elements of Quality Online Education and the Five 
Pillars from the Sloan-Consortium (Bourne & Moore, 2003, 2004; Moore, 2005). For this 
research, the Institute for Higher Education Policy (IHEP) Quality on the Line study was one of 
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the first documents of its kind that delineated common quality indicators specifically for online 
education and the Midwest University was one of the six institutions selected nationally for 
further investigation in this case study (Phipps & Merisotis, 2000). 

Finally, quality monitoring within the STEM fields extends from the “quality 
management” traditions, which originated in manufacturing because of the desire and need to 
monitor product quality resulting from production increase and worker decrease during World 
War II (Bennett, 1996; Merna, 2008). Several engineering organizations formed an accreditation 
body known as the Engineers Council for Professional Development that became the 
Accreditation Board for Engineering and Technology (ABET) that is now known by the 
acronym (ABET, 2009; Prados et al., 2005). The ABET sought to revamp the accreditation 
criteria to emphasize nine criteria known as the Engineering Criteria 2000. Educational 
institutions and programs voluntarily undergo a review every six years to determine if these nine 
criteria are attained (ABET, 2009; Prados et al., 2005). Most postsecondary undergraduate 
programs, including the programs at Midwest University in agriculture and engineering, comply 
with the ABET requirements. While these nine criteria are equally important to the overall 
process, ABET’s criterion three (program outcomes and assessment) is often cited as the most 
challenging and important in program quality monitoring and subsequent accreditation (Dick & 
Deborah, 2003; Kwok-Bun, 2007) and is highlighted as important for graduate programs and 
online programs (Bourne et ah, 2005). 

While all or some of these quality monitoring systems are commonly understood and 
explored in the literature, the way to evaluate the attainment of these quality indicator categories 
has not been fully developed. Most scholars report a dire need for improved evaluation and 
assessment, particularly in online learning, because few higher education institutions have 
written guidelines or policies on online learning (Law, Hawkes, & Murphy, 2002; Rovai, 2003; 
Tallent-Runnels et ah, 2006). Law and associates contend, “While many researchers describe 
similar outcomes for online classes and, by extension, entire online degree programs, their 
analyses often call for further investigation and development of a clearly articulated set of 
criteria for consistent evaluation across programs” (Law et ah, 2002, p. 83). Further, this lack of 
attention to program evaluation and quality indicator development within higher education is one 
of the key factors which contributes to the failure of some online programs nationally and 
globally (Rovai & Downey, 2009). However, little study has been done about what programs 
like Midwestern University are really doing to discern the quality of their programs through 
evaluation, especially since Midwestern University was actively involved with the NRC study, 
IHEP study, and ABET accreditation. 

METHODS 

This study employed a multiple case study design using a holistic approach (Yin, 2002). 
The case study approach seemed appropriate since the research team wanted to probe “how and 
why” questions(Stake, 1995; Yin, 2002). This case study included a combination of document 
review, surveys (n = 107), and interviews (n = 27) with program administrators, faculty, and 
students from the three established STEM online degree programs at Midwestern University. 
Data were collected and analyzed in sequence during the late fall of 2009 and the spring, 
summer, and fall semesters of 2010. First, a comprehensive document review with interviews 
with the program administrators were conducted and analyzed to investigate current evaluation 
practices. The second phase included administering a survey to online students and faculty to 
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gather information about the presence and importance of these practices and quality. Faculty and 
student interviews were then conducted. The administrator survey was given to probe current 
practices and to gauge administrators’ feedback on the quality indicators. 

SAMPLED PROGRAMS 

In order to be a part of this study, programs needed to meet six criteria, including being 
(a) an online degree program affiliated with the Midwestern University in the fall of 2009; (b) a 
complete Master’s Degree Program; (c) an established program housed in a department for at 
least five years at the main campus at Midwestern University; (d) one of the nine STEM 
departments identified by the National Science Foundation; (e) an online program where all of 
the content is delivered online, exceeding the requirement as defined by the Sloan Consortium, 
which was that at least 80% of the content be delivered online, and (f) a program agreeing to 
participate in the study with the support of the department head. As a result of these criteria, 
three STEM online Master’s degrees were included in the study: the Master of Science in 
Natural Resources and Environmental Sciences in Department of Natural Resources and 
Environmental Sciences (NRES), the Master of Crop Sciences in the Department of Crop 
Sciences (CRSC), and the Master of Mechanical Engineering in the Department of Mechanical 
Science and Engineering (MechSE). 

All three of the programs require zero on-campus visits before or during the coursework 
to complete the degree. However, the degrees through the College of Agricultural, Consumer, 
and Environmental Sciences (ACES), including the NRES and the CRSC degrees, require a one- 
day onsite visit for a final oral exam at the completion of the program. The MechSE degree 
through the College of Engineering required no visits for any of the degree requirements. The 
fall and spring course terms last 15 weeks, which is the same time duration as the traditional 
courses delivered at the Midwestern University campus. Also, these courses occur 
simultaneously with the on-campus programs (beginning in late August for the fall semester and 
in January for the spring semester). The programs require 32 credit hours for completion of the 
Master’s degree. Admission prerequisites include an earned bachelor’s degree from an accredited 
institution, as well. 

Both online departments present information in different ways to varying numbers of 
students each semester. In engineering, lectures are streamed online via a dedicated engineering 
and computer science online portal. In the previously taped video, the professor is at the front of 
the room at a podium lecturing to a live class. The information, curriculum pacing, and 
presentation are touted as being “identical” to what an on-campus student is receiving. In the 
online setting, the lecture screen with the lecturer is in the right- or left-hand margin, while the 
lecture slides can be viewed individually or all at once (see Figure 1 in Appendix). Each 
engineering class serves from two to sixteen online students during the fall, spring, and summer 
sessions. According to the program administrator, there were 74 to 75 students enrolled in the 
program. When asked about graduates, the program coordinator said they did not have this exact 
information available, but approximately eight had graduated from the program since 2007. 

In ACES, the lectures are streamed live online, meaning the lecture material is produced 
one time per week each semester by the instructor of the course. This process is facilitated by a 
commercial company called Elluminate Live, which is a part of Blackboard, Incorporated. 
Lectures are synchronous, meaning that the instructor delivers a live lecture, and then students 
are allowed to pose questions via personal microphones and chat tools (depicted in the training 
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slide in Figure 2 in Appendix). Moreover, a synchronous section is available for posting 
questions as well, and these sessions are all archived and able to be retrieved within the system. 
Lecture notes and other course information are then posted and accessible to students via the 
Midwestern University course management system known as Midwestern University Compass 
(see Figure 3 in Appendix). The courses in ACES serve 5 to 15 students at one time. According 
to the program administrator, CRSC currently has 48 admitted students and NRES has 86 
students. Over the last five years, 26 students have graduated from CRSC and approximately 15 
have graduated from NRES. 

RESULTS 

According to the administrators and faculty of the program, limited evaluation is 
occurring in the different programs. In fact, when the researcher requested any evaluation 
documents and information collected by the programs, one administrator emailed the following 
response, “We don't do any [formal] evaluation at this time of our online programs that you are 
looking at. The departments probably need to do one.” Moreover, and they were vaguely aware 
of the fact that the Midwestern University had participated in the NRC study and ABET 
accreditation process but the programs did not know about the IHEP online criteria and had not 
used any of the data from these studies. Suggesting that what is ideally stated in the literature, 
might not be filtering to the some programs. Three key themes related to current evaluation 
practice emerged as a result of analysis of the survey data, interviews, and document review 
process. These included the use of: (a) informal feedback from the students and faculty, (b) 
student satisfaction surveys (i.e., ICES student feedback and department-specific and created 
satisfaction surveys), (c) student grades and performance information. 

Informal Student and Faculty Feedback 

A majority of evaluation occurring in both programs was identified as “informal 
feedback” (i.e., questions, praise, and/or suggestions by non-formal measures like email or in- 
class feedback) given by the students to the faculty and administrators and by the faculty to the 
administrators of the program. The two administrators were asked to rate the number of times 
they used certain information to evaluate their programs (depicted in Figure 4 in Appendix). 

Each bar represents the responses from each administrator (i.e., when the bar has two colors, the 
administrators indicated different levels of use of the particular evaluation source). The twofold 
purpose of this question is the fact that different program administrators reported some similar 
practices, but the practices were being implemented at varying levels. The most-often-cited 
categories were the informal feedback from instructors and students (both with means of 4.00 
and SD of 1.41), with one administrator indicating that she used each of these “sometimes” 
(meaning three to five times per year, represented by the green bar half on the chart) and the 
other indicating that she used this “almost always” (meaning six or more times per semester, 
represented by the light blue). 

Analysis of the open-ended survey item which asked administrators to identify the three 
most worthwhile evaluation activities they do to evaluate their programs revealed this same 
trend. The first two responses from both administrators included this notion of informal feedback 
from students and faculty. One administrator reported that two of the important evaluation 
activities were “speaking with students directly and speaking with faculty directly.” The other 


Evaluation of STEM, Page 6 



Journal of Case Studies in Education 


reported that she “listen[s] to current students when they give unsolicited input,” and that she 
“listen[s] to current faculty members when they provide input.” 

Interview data from administrators also supported this assertion about “informal 
feedback” being a primary source for evaluating the quality of the courses, instructors, and 
overall program. One reported, “We look at [the program], but very, very informally. It’s all very 
informal. At the end of the year, [the assistant director] and I will look at and talk about: ‘Is this 
faculty member happy? Does he need anything? Is he wanting to teach for us again? If not, 
should we talk to him about why not?’ It’s more of a time issue; we kind of deal with the issues 
that we see and hear about, but that’s all.” 

Student Satisfaction Surveys 

It was also found that the program coordinators and faculty are looking at more formal 
evaluation data from student satisfaction surveys in order to discern quality of their online 
programs. The student satisfaction surveys referenced were divided into two primary categories: 
(a) surveys from the Instructor and Course Evaluation System (ICES) used by all courses (both 
traditional and online) at Midwestern University and (b) program-produced surveys that are 
reportedly more applicable to the particular program where the online degree is housed. 

The ICES process is facilitated through the Center for Teaching Excellence (CTE) at the 
Midwestern University. The most widely used “Course Evaluation Questionnaire” consists of 21 
general concept questions rated on a five-point scale (from “strongly agree” to “strongly 
disagree”) and four open-ended questions. Among the 21 multiple choice items, students are 
asked to rate: the instructor’s overall teaching effectiveness and their level of interest, 
engagement, presentation style, and material knowledge; the course’s overall quality, including 
the course content, degree of difficulty, and structure; and the student’s self-motivation and 
preferences on teaching method. The four open-ended items include questions asking the 
students to comment on the major strengths and weaknesses of the instructor, the course aspects 
that were the most beneficial, improvement suggestions, and the grading procedures and exams. 
Findings regarding student satisfaction surveys and, specifically, use of the ICES for evaluative 
purposes were reported by the administrators and faculty via their surveys and interviews, but it 
varied by program. First, for administrators, ICES use was variable primarily because ICES 
ratings are only available to program administrators if they are shared by the faculty. One 
administrator summarized this use by saying, “ICES has always been in existence; we have 
always used ICES when we can.” 

Faculty also cited ICES surveys as a frequently-used tool that they used to generate 
evaluation information about the quality of the program. All interviewed faculty reported that 
survey data collected at the end of the semester was a mechanism that they use to evaluate their 
courses. A majority (n = 7) of the faculty reported in the interviews that the only survey their 
students completed for the online program was the ICES and they had no knowledge or 
recollection of the programs generating any other surveys. 

Of the 16 faculty who submitted answers on the open-ended survey question regarding 
the three most worthwhile activities that they as faculty members do to evaluate the quality of 
their course and program, six mentioned ICES and/or student surveys. One reported that he looks 
at “feedback from students on ICES,” while another said, “use of a special course evaluation 
survey” was a worthwhile activity. The administrators from one program tried to share the 
program-generated survey findings with faculty, but this had been unsuccessful, and they had 


Evaluation of STEM, Page 7 



Journal of Case Studies in Education 


since just generated this information for internal use. She said, “One time we shared one of those 
course evaluations, and he tore it into pieces, and we never gave it to them again.” Additionally, 
two faculty members recorded that they generate their own customized surveys to gather student 
feedback. One said, “The online surveys I create and distribute (via Google apps).” Therefore, it 
appears that although faculty and administrators are reporting use of this information for 
evaluative purposes, the data suggest that there is not a uniform approach, form, or reporting of 
this information within or between the departments. 

Student Grades and Course Assessments 

The third category of current evaluation practice for discerning program quality reported 
by both program administrators and faculty was related to the monitoring of student grades and 
course assessments. While these stakeholder groups acknowledged using this information, 
interviewees revealed that online students’ achievement scores during the program on both the 
course assessments and course completion grades appeared to be a source of tension between the 
program and different stakeholders. 

For administrators, contradictory reports were sometimes given about use of these data. 
First, evident from Figure 4 above, one administrator reported using each source—student grades 
and student performance on course assessments—“very few times” (meaning one to two times 
per year, represented by the red half of the bar on the chart), and the other administrator reported 
using this source “almost always” (meaning six or more times per semester, represented by the 
light blue half of the bar). According to interview data, contradictory statements were made 
regarding these data and the use of these data. For instance, one administrator reported, “Grades 
is another way [we evaluate quality]; we do look at the average GPA of our students when they 
graduate.” The other administrator stated: 

So we don’t really look at grades. Our goal is not to transform education, which is a great 
goal, but it’s not our goal. Our goal is to make sure that online students get the exact 
same course they would get if they were here on campus. That’s really what we are doing 
[to evaluate quality], and that’s really what our students expect. Our courses are really, 
really hard, and they are also theoretical, and a lot of the working professionals want 
practical courses. We are a research university; our courses are going to be theoretical. 
[The professors] evaluate their students’ exams; that’s their realm that we don’t get into. 
So ours is on the students’ satisfaction. 

However, later in the interview, the administrator contradicted herself and said that the 
program references grades more so for a check on a newer professor rather than the student. She 
said, “If we know there have been grading or communication issues, we emphasize those things 
again [with the new professors]—the things we would like for them to know about. In one case, 
we have had to go to the department and ask that this person not teach anymore for our program, 
and they honored that. That has only happened once.” 

Related to course assessments, one program administrator talked about the thorough 
process they use to keep the integrity of the examination documents and process by ensuring that 
each student has a proctor. The consistency of this process was credited as an indicator of the 
quality of their program. She reported: 

A lot of our operation here centers around exam administration. Because they are 
[department] courses, we ask the instructor to bring a copy of the exam over a week in 
advance of the date [on] which it is supposed to be given on campus.. .They are paper and 


Evaluation of STEM, Page 8 



Journal of Case Studies in Education 


pencil exams, and they are typically timed, usually closed book, closed notes. You can 
only have a calculator; you can’t really have your cell phone, and you can’t have this 
other stuff. Because of that, that necessitates proctors. At the beginning of every 
semester, every student is required to submit that name of someone whom they are 
proposing as their proctor, and there is a list of criteria. They can’t be a peer; they can’t 
be someone related to them; it has to be a manager or somebody in the HR training 
department. If there is more than one student with a company, they have to take the exam 
at the same time. The integrity of the exam is important to the program to keep that 
integrity intact, so that’s why we go to these lengths... The process is important to the 
course. 

Faculty also cited student performance on the course assessments and overall as a tool 
that they frequently used to generate evaluation information about program quality. Ten of the 
eleven faculty interviewed reported using student performance as a mechanism to determine the 
quality of their courses. One said, “I use my teaching experience, since it takes experience to 
judge how successful [the course] is. I mainly look at the performance of the students as [to] 
whether they are successful. Generally, some have been successful, and others haven’t, and it’s 
about their performance.” Another said, “If most of my students put in the time and do well, then 
it is fine.” However, when the interviewer asked what happens if the students, and particularly 
online students, do not perform well, the professors generally encouraged students to “review the 
material.” One reported, “The goal is to make this course as good as you can in the best possible 
way in teaching online. All the students get access to the material, and I send uniform emails to 
the students. So the [online] students are not at a disadvantage. Everyone has access to the web 
pages. The online course and on-campus lecture is exactly the same material. I tell them to check 
the material.” 

Of the 16 faculty who submitted survey responses to evaluation activities, six mentioned 
course performance; three referenced a specific time or assessment, while the other three were 
more comprehensive for the course as a whole. For instance, one reported, “I study the students’ 
performance on the final exam and how well they are keeping up with the online assessments as 
[the course] is progressing.” Another faculty member reported he does a “periodic review of the 
student learning outcomes and grades,” while another said he looks at “the quality of the term 
papers and the assessments through the same on-campus offerings of the same material.” 

DISCUSSION 

This case study focused on developing a deeper understanding of how three online degree 
programs evaluated quality of their respective programs. Magjuka, Shi, and Bonk assert that 
there are ten major considerations for administrators who are designing programs that are 
associated with online education (Magjuka, Shi, & Bonk, 2005); however, overall assessment or 
evaluation of the program was not a topic that these researchers raised. Previous studies suggest 
that intuitional expectations of looking at formal student evaluations and feedback can influence 
both faculty and students’ perceptions of judgments of quality and the use of this information for 
traditional programs (Stein et al., 2013). Conversely, there is limited literature that talks about 
the informal evaluative information in online education that was cited by the participants in this 
study. However, the current case study also uncovered several issues with the findings above 
and are outlined below consistent in regards to the use of: (a) informal feedback from the 
students and faculty, (b) student satisfaction surveys (i.e., ICES student feedback and 


Evaluation of STEM, Page 9 



Journal of Case Studies in Education 


department-specific and created satisfaction surveys), (c) student grades and performance 
information. 

Issues with Informal Feedback 

While both programs and administrators acknowledged the heavy use of informal 
feedback for evaluating the quality of their online programs, there were obviously issues with 
using this. Issues revealed from the data analysis showed that using informal student and faculty 
feedback: lacks structured collection and reporting mechanisms and it benefits the on-campus 
students more so than the online students in some cases. 

Lacks structured collection and reporting mechanisms. While both administrators 
acknowledged that informal feedback is currently used as the primary source to evaluate the 
quality of the program, no structured mechanisms are in place to track the collection or 
resolutions involving “informal feedback” administrators get from either faculty or students. One 
administrator said they have a blog for questions, but this is primarily for overarching, general 
questions on topics like graduation and registration procedures and not as much for feedback on 
courses. Related to this topic she said: 

We have a blog set up, for example, where students can blog and ask questions so that 
everybody can see the answer so.. .1 don’t have to repeat them 50 thousand times. We 
don’t want the quality of service to diminish, so we are trying to figure out some ways [to 
document this].” 

This trend of self-reported use of “informal feedback” for evaluation was found from the 
faculty of the program as well. Ten of the 11 faculty interviewed reported that they specifically 
use “informal student feedback” in order to inform their instruction and to monitor student 
understanding. After one interview, one faculty member shared an example of an email as a 
sample of this informal feedback. The instructor’s email said, “Your study came to mind as I 
responded to the [online] student in the email exchange... While all of the details may not be 
apparent from the exchange, it’s the kind of minor ‘success story’ that I look for as [informal 
student] feedback when I teach (online or on campus).” 

Benefits the on-campus students more so than the online students. While faculty reported 
that they used the informal feedback from the students to influence their presentations and as 
their “gauge” for student understanding, further probing of this topic during interviews revealed 
that this was primarily true for their on-campus traditional classes, but not necessarily so for their 
online courses. This seemed especially true for those six faculty in the program where taped 
lectures were streamed online; this reported informal feedback came more directly from students 
in the on-campus program rather than the online program. Those instructors who had their 
lectures taped, sometimes three to five years prior to the streaming of the class, reported that they 
used the informal feedback to change the lectures, but when further probing was done, it turned 
out that these changes were made for students in the traditional environment and were not 
reflected in the lectures for the online students. Upon this admission, one faculty member stated, 
“I always said that I’m doing it, but it is evident for me [in] talking with you that I’m just doing 
this for the on-campus students. This is an ‘ah-ha!’ moment for me.” Another faculty member 
also stated: 
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I am not doing this nearly enough. I’m not retrospective about my lecture about what 
needs to be improved. I know my face-to-face time with my [on-campus] students is 
different because my office door is opened, and they’re free to come in. On a phone, it’s 
not quite the same. I ask about the ‘muddiest’ point and collect cards on this, but this 
does not work well with online courses. 

While weekly student question sessions were held for both online programs, and the 
instructors reported that they would clarify any issues raised or questions posed by the online 
students during that time, changes made in the traditional class session were not made in online 
classes unless online students raised the same questions that the traditional class had in the face- 
to-face sessions. In other words, content changed or further explained in the face-to-face, 
traditional class setting might be different than the information shared with online students, and 
vice versa. This admission seemed to bother a few of the interviewed faculty as they came to the 
realization that they were not evaluating their on-campus programs in the same way. One said, “I 
have never thought about [these] changes in this way. I say it’s the same, but I guess there are 
differences in the changes that happen. The on-campus students have full access to everything 
that I do with the online students to make sure that they are getting the same support from the 
lectures, but I don’t know that I’ve done enough to make sure that the online students get the 
changes [made in the lectures].” 

Issues with Using Student Satisfaction Surveys. 

Although ICES forms are administered each semester for almost all of the courses 
(including both traditional and online courses) in the Midwestern University system, 
administrators reported that ICES scores were used sporadically in the online programs in order 
to evaluate quality. Due to the fact that the ICES ratings are only available to college and 
program administrators if the faculty member chooses to have these released before or after 
student ICES ratings are calculated, the amount of use reported by the program administrator 
was variable. Evident from Figure 4 above, one administrator indicated that she used ICES 
scores “very few times” (meaning one to two times per year, is represented by the red half of the 
bar on the chart), and the other indicated she used this source “almost always” (meaning six or 
more times per semester, represented by the light blue half of the bar). One administrator 
reported that she directly tells faculty to share these with her every semester, while the other said 
that she was cautioned against doing this by the department head. 

Neither of the administrators reported using ICES in the opened-ended survey items; 
however, both interview transcripts had instances of this being noted. One administrator 
acknowledged using the ICES when it’s available, but reported that differences in their program 
had required them to create their own survey for one of their programs. This had caused some 
tension in the program, because not all of the online programs in the college or instructors 
wanted to do this different form. She said, “I only get [the ICES] scores if they give them to me. 
So we have those [program-produced] evaluation forms, and [another online program] has them 
do the ICES because they have that system set up in place. I think at some point, the whole 
university is going to have to do ICES, but online courses right now are all different.” She also 
reported that their office had produced overviews of the student satisfaction data from the 
program-produced evaluation forms, and faculty were very apprehensive about the validity of the 


Evaluation of STEM, Page 11 



Journal of Case Studies in Education 


findings. Additionally, the program coordinator had full access to these findings, thereby taking 
the faculty choice for disclosing the information out of their hands. 

Issues with Using Student Grades and Course Performance 

While the administrators and some faculty reportedly considered the sources of student 
grades and assessment performance to be an indicator of quality, some professors in the program 
stressed that the quality of the students at times rendered these marks useless for assigning 
program quality. Three faculty recorded survey responses related to the quality of the students, 
and therefore these students’ grades, as a major stumbling block to the quality of the program. 
One wrote, “It’s unclear what admission standards online students have met and where they are 
in a given program. Some struggle.” Another faculty member wrote, “Online program/courses 
need to be marketed better to attract higher quality applicants. Many online students have been 
away from school for many years, and it is a shock when they see math and physics and fail.” “If 
the student quality is low, then their performance and reaction to the course cannot be used to 
evaluate it,” said the other. Administrators also acknowledged this tension, but primarily 
deflected this as a problem that had been rectified by recruiting efforts. One reported: 

The qualifications of our students—the level and the quality of the students—is 
increasing every year.. .we are getting more and more students in [the online] programs. 
[Department name] probably had a little lower of a benchmark for some of the students 
coming in in previous years, and that has been a complaint of the department faculty 
members. But the faculty members that have complained about that don’t work in the 
department, and so they don’t know the quality of the students, and the faculty members 
that do work in the program say, “Hands down, I would put those students against my on- 
campus students, and they would perform as good or better any day of the week,” and 
that’s what they’ll tell me. 

The other administrator also acknowledged this problem. She said: 

We had a relationship with [Company Name], and what we did was kind of a marketing 
thing with them which if they sent in a certain number of students, then we would rebate 
them a certain percentage of tuition. That went horribly wrong. It did sound good, but it 
turns out that because their students knew we had this agreement with them, they thought 
they could do whatever they want, get whatever grades, and we still owed them a degree. 
I wasn’t involved in the negotiations, but that had nothing to do with the negotiation 
agreements. So that was really tough, but that [agreement] is gone now.” 

Further, using grades as an evaluation is problematic from the students’ standpoint as 
well. For instance, three faculty members said that a strength of the online program is that it 
offers additional curricular support for the on-campus students, who have full access to the 
online lectures and material. One faculty member said, “I have had [on-campus students] watch 
the online lecture if I had to travel out of town for a conference, so my schedule is not 
interrupted.” Thus, in some cases, the addition of an online component enabled traditional 
students to not only have access to the traditional classroom information, but also to online 
support as well. These extend beyond the hybrid classes or blended-leaming environments, 
where roughly 30% to 79% of the course is delivered online (Allen & Seaman, 2008). This 
allows on-campus students to have the “benefits” of both the traditional and online setting at the 
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same time, if the students choose to participate. This is especially problematic in courses where 
grades are assigned on a curve for the on-campus and online students. In the interviews, three 
students cited this as a problem they had with the program. One student reported, “You get an 
exam back, and you get an 86, and this may be a low A or B, but because the lectures are taped, 
we don’t know where we are. You don’t get a fair estimate of where you stand. The grades in 
these classes are more based on how well the class did. [The on-campus students] get our 
information and the on-campus information, and they can have a better idea about what their 
grade really is because they can talk to people in the class.” 

CONCLUSION 

It is important to note that despite the administrators and faculty of these online graduate 
programs citing that informal feedback was their primary means of evaluation, their often 
admissions of the issues of this with further probing in the case study proved enlightening to this 
research. Therefore, it is important to note that continued discussion and study of this topic with 
well-established online programs like those at Midwestern University and other programs is just 
as important as offering frameworks and “advice” for evaluating newer programs. 

Further, it is essential to consider that evaluation and assessment is continuing to be 
elusive and essential in the ongoing success. As previously stated, establishing common 
indicators of quality between online and traditional environments and ways to probe this notion 
of quality is a necessity because of the trepidation among some in higher education that online 
education is jeopardizing the quality of higher education (Burbules & Callister, 2000; 
Larreamendy-Joerns & Leinhardt, 2006; Rovai, 2003). Based on the findings from this present 
case study, it is clear that there is an ongoing need to explore this as online learning programs 
might still be relying on informal feedback. 
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APPENDIX 


Engineering and Computer Science Onlir 


Lecture Notes View All Slides Email Feedback Help 



resh Table of Contents 


TAM 542 Micromechanics of Materials 

Lecture 1 

Overview 

& 

Review of Basics of Solid Mechanics 


Figure 1. An example of an online lecture in engineering 



Figure 2. An example of an ACES online communication mechanism provided by the 
Elluminate program. 
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Figure 3. An example of an ACES course management system provided by the Midwestern 
University Compass program. 
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Q45:dd. Informal feedback from instructors 
Q46:ee. Informal feedback from students 
Q16:a. Instructor and Course Evaluation System (ICES) forms 

Q17:b. Student Grades 
Q18:c. Student Performance on Course Assessments 
Q43:bb.CEEED results (on the year of participating in a review process) 
Q32:q. Course sequences and offerings 
Q33:r. Course enrollment 
Q36:u. Expenditures and Revenue 

Q41:z. Company feedback who employ the students (i.e., topics like... 
Q22:g. Student satisfaction surveys (program produced) 
Q28:m. Course completion 

Q44:cc. CEEED results (NOT on the year of participating in a review... 

Q29:n. Degree progress 
Q31:p. Persistence/retention rates 
Q34:s. Course structure and format 
Q40:y. Readmission and registration procedures 
Q24:i. Course usability reviews 
Q25:j. Course accessibility reviews 
Q23:h. Instructor surveys (program produced) 
Q26:k. Demographics of students 
Q30:o.Timeto degree averages 
Q35:t. Security issues 
Q37:v. Prospect status 
Q27:l. Student Career Goal Statements 
Q38:w. Alumni Feedback 
Q20:e.Lead Generation Information Software 
Q21:f. Finiancial aid availability 
Q19:d. U.S. News and World Report Rankings 
Q39:x. Diversity of faculty 

Q42:aa. Achievment information of faculty (i.e., honors, grants, and... 

■ Notat all (0 times) ■ 

■ Sometimes (3*5 times per year) i 

■ Almost always (6 or more times per semester) i 



Very few times (1-2 times per year) 

Often (3-5 times per semester) 

Not applicable (this information is not collected or available) 


Figure 4. Administrator responses to survey item regarding the frequency that they use the listed 
data sources to evaluate their online program’s quality, sorted by averages. Each bar represents 
two responses. 
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